Highly Parallel Computers for Artificial Neural Networks

ii I do not know what I may appear to the world, but to myself I seem to have been only like a boy playing on the seashore, and diverting myself in now and then finding a smoother pebble, or a prettier shell than ordinary whilst the great ocean of truth lay all undiscovered before me. ABSTRACT iii ABSTRACT During a number of years the two fields of artificial neural networks (ANNs) and highly parallel computing have both evolved rapidly. In this thesis the possibility of combining these fields is explored, investigating the design and usage of highly parallel computers for ANN calculations. A new system-architecture REMAP (Real-time, Embedded, Modular , Adaptive, Parallel processor) is presented as a candidate platform for future action-oriented systems. With this new system-architecture, multi-modular networks of cooperating and competing ANNs can be realized. For action-oriented systems, concepts like real-time interaction with the environment , embeddedness, and learning with self-organization are important. In this thesis the requirements for ef ficient mapping of ANN algorithms onto the suggested architecture are identified. This has been accomplished by studies of ANN implementations on general purpose parallel computers as well as designs of new parallel systems particularly suited to ANN computing. The suggested architecture incorporates highly parallel, communicating processing modules, each constructed as a linear SIMD (Single Instruction stream, Multiple Data stream) array, internally connected using a ring topology, but also supporting broadcast and reduction operations. Many of the analyzed ANN models are similar in structure and can be studied in a unified context. A new superclass of ANN models called localized learning systems (LLSs) is therefore suggested and defined. A parallel computer implementation of LLSs is analyzed and the importance of the reduction operations is recognized. The study of various LLS models and other commonly used ANN models not contained in the LLS class, like the mul-tilayer perceptron with error back-propagation, establishes REMAP modules as an excellent architecture for many different ANN models, useful in the design of action-oriented systems. Highly Parallel Computers for Artificial Neural Networks iv PREFACE v PREFACE This thesis deals with the implementation of artificial neural networks on massively and highly parallel computers. The thesis consists of nine papers. The first two papers introduce the concept of noncon-forming massively parallel computers and survey parallel computer architectures used for artificial neural networks. The two following papers are descriptions of the REMAP architecture which I use as …